Method and a system for determining radiation sources characteristics in a scene based on shadowing analysis

ABSTRACT

A method and a system for analyzing distinct light or radiation sources affecting a scene are provided. The method may include: sensing at least one image of a scene containing surfaces and objects, wherein the scene is illuminated by at least one distinct radiation or source; maintaining a database of the scene which stores approximate positions of at least some of the objects in the scene; identifying at least one candidate silhouette suspected to be cast by the at least one distinct light or radiation source, deriving properties of the at least one distinct light or radiation source, based on the at least one identified silhouette, based on data derived from the database.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 15/901,402, filed Feb. 21, 2018, which will issueas U.S. Pat. No. 10,395,135 on Aug. 27, 2019, which in turn is acontinuation application of U.S. patent application Ser. No. 14/947,584,filed Nov. 20, 2015, issued as U.S. Pat. No. 9,928,441 on Mar. 27, 2018,both of which are incorporated by reference herein in their entirety.

FIELD OF THE INVENTION

The present invention relates generally to the field of imageprocessing, and more particularly to determining wave sourcescharacteristics in a scene based on shadowing analysis carried out on acaptured scene.

BACKGROUND OF THE INVENTION

Prior to the background of the invention being set forth, it may behelpful to set forth definitions of certain terms that will be usedhereinafter.

The term ‘sensing device’ as used herein is broadly defined as anycombination of one or more sensors that are configured to capture 2Dimages of a scene. The sensing device may include any type of passivecamera or sensor array that can capture radiation and derive an imageout of it (examples may include: visible light, IR, ultra sound, radarsensor, and the like).

The term ‘shadow’ as used herein is broadly defined as a region whereradiation from a radiation source (such a light source) is obstructed byan opaque (in the appropriate wavelengths) object. The shadow occupiesall of the three-dimensional volume behind an object with light orradiation in front of it. The cross-section of a shadow is atwo-dimensional silhouette, or a reverse projection of the objectblocking the radiation or light.

The term ‘point light source’ or ‘point radiation source’ as used hereinis defined as a light source or a radiation source that only casts asimple shadow, called an “umbra”.

The term ‘non-point light source’ or ‘extended light source’ or‘extended radiation source’ as used herein is defined as a radiation orlight source whose shadow is divided into the umbra, penumbra andantumbra. The wider the light source or radiation source, the moreblurred the shadow becomes. If two penumbras overlap, the shadows appearto attract and merge.

The term ‘ambient light’ or ‘ambient radiation’ as used herein isdefined as light or radiation that is diffused beyond a certain level sothat the outlines of shadows are soft and indistinct or completelynon-existent. For example, the lighting of an overcast sky produces fewvisible shadows.

The term ‘distinct light source’ or ‘distinct radiation source’ as usedherein in the context of the application is a residual definition of theambient light discussed above. Generally speaking, a distinct radiationlight source will cause objects in a scene to cast shadows havingdistinct outlines. The outlines of the shadow zones can be found bytracing the rays of light emitted by the outermost regions of theextended light source. If there are multiple light sources, there willbe multiple shadows, with overlapping parts darker, and variouscombinations of brightness levels or even colors.

One of the challenges of computer vision is to detect the presence, andobtain knowledge about distinct radiation or light sources (as opposedto ambient light or ambient radiation) in a scene. The importance ofradiation light source analysis is crucial in some use cases such asaugmented reality applications in which virtual objects are insertedinto the scene and it is essential that the virtual objects will beaffected by the lighting conditions as any other real object present inthe scene. Additionally, it is also important for enhancing the illusionnotion of the viewer, that the virtual objects will cast shadows in asimilar way to real objects, given the specified light or radiationconditions.

It would be therefore advantageous to suggest some logic or a flow thatwill enable a computerized vision system to determine the properties ofall distinct light or radiation sources that affect a specific scenethat is being captured and analyzed.

SUMMARY OF THE INVENTION

Some embodiments of the present invention include a method and a systemfor analyzing distinct light sources affecting a scene. The method mayinclude: sensing at least one image of a scene containing surfaces andobjects, wherein the scene is illuminated by at least one distinct lightor radiation source; maintaining a three-dimensional database of thescene which stores three-dimensional approximate positions of at leastsome of the objects in the scene; identifying at least one candidatesilhouette suspected to be cast by the at least one distinct light orradiation source, deriving properties of the at least one distinct lightor radiation source, based on the at least one identified silhouette,based on data derived from the three-dimensional database.

These, additional, and/or other aspects and/or advantages of the presentinvention are set forth in the detailed description which follows;possibly inferable from the detailed description; and/or learnable bypractice of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features, and advantages thereof, may best beunderstood by reference to the following detailed description when readwith the accompanying drawings in which:

FIG. 1A is a block diagram illustrating a system in accordance with someembodiments of the present invention;

FIG. 1B is a block diagram illustrating a system accordance with otherembodiments of the present invention;

FIG. 1C is a block diagram illustrating a system accordance with yetother embodiments of the present invention;

FIG. 1D is a block diagram illustrating a system accordance with yetother embodiments of the present invention;

FIG. 2 is a drawing illustrating an aspect of a system in accordancewith embodiments of the present invention; and

FIG. 3 is a high level flowchart illustrating non-limiting exemplarymethod in accordance with embodiments of the present invention;

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, various aspects of the present inventionwill be described. For purposes of explanation, specific configurationsand details are set forth in order to provide a thorough understandingof the present invention. However, it will also be apparent to oneskilled in the art that the present invention may be practiced withoutthe specific details presented herein. Furthermore, well known featuresmay be omitted or simplified in order not to obscure the presentinvention.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing,” “computing,”“calculating,” “determining,” or the like, refer to the action and/orprocesses of a computer or computing system, or similar electroniccomputing device, that manipulates and/or transforms data represented asphysical, such as electronic, quantities within the computing system'sregisters and/or memories into other data similarly represented asphysical quantities within the computing system's memories, registers orother such information storage, transmission or display devices.

FIG. 1A is a block diagram illustrating an exemplary architecture onwhich embodiments of the present invention may be implemented. System100 may include a sensing device 110 configured to capture a scene thatmay include objects (e.g. 10). System 100 may further include a storagedevice 120 configured to maintain a database of the scene which storespositions of at least some of the objects in the scene (including, forexample, object 10). System 100 may further include a computer processor130 configured to identify at least one candidate silhouette 12suspected to be cast by the at least one distinct light source 14, andderive properties 134 of the at least one distinct light source 14,based on the at least one identified silhouette, based on data derivedfrom the three-dimensional database 120.

According to some embodiments, sensing device may include any type ofpassive camera or sensor array (either visible light, IR, and the like).The 3D data about the scene stored on 3D database 120 may be partial innature and may be provided in advance or derived online as the locationof the objects may change over time. Alternatively the 3D data may bederived by system 100 for example using moving sensing device 110 withcomputer processor 130 configured for mapping the scene.

Light or radiation source properties 134 of distinct light or radiationsource 14 may include:

-   -   Location in 3D space (in some fixed coordinate system);    -   Structure—for example: point light source (bulb); point        radiation source (omnidirectional RF antenna) line light source        (fluorescent); ambient lighting or radiation (no direct light or        radiation source, this is the “average” light or radiation in        the scene); Other light or radiation sources can be seen as a        combination of points and lines sources;    -   Color (wave frequency composition, RF bandwidth);    -   Luminance; and    -   Directionality (e.g., spotlight, antenna directivity in RF        radiation).

According to some embodiments, the properties of all or some of thelight source affecting the illumination conditions of the scene can beused for several use cases as detailed below:

-   -   When adding virtual objects to the scene, to render shadows that        match the lighting or radiation of the scene;    -   When trying to simulate how an environment would look like when        the lighting or radiation changes (i.e., interior design).

According to some embodiments, detecting a stationary silhouette (seenin all frames) is carried out by using segmentation techniques. Objectsthat should have the same color and/or texture are detected. Then placeswhere the overall brightness (or of only some color components) issmaller and gradually increases, are analyzed as an indication of asilhouette.

According to some embodiments, detecting a dynamic silhouette (when someviews of the area contain the silhouette and some without) can becarried out by knowing the position and orientation of the camera, and3D structure of the scene, and recognizing this is the same area withdifferent color/brightness in different points in time (there might be aneed to adjust to camera white balance/exposure/gain/and the like). Whenthe change is more than a pre-defined threshold, this area is acandidate to be a silhouette.

FIG. 1B is a block diagram illustrating a system in accordance withother embodiments of the present invention. In addition to allcomponents shown in FIG. 1A, the silhouette here is a compound oneconstituted of three areas 12A, 12B and 12C being darker than 12A, 12B.This implies two light or radiation sources, for each one, object 10casts a shadow. In accordance with embodiments of the present invention,the compound silhouette is segmented into the silhouettes generated by asingle light source and for each one of them, the properties of thecorresponding light or radiation source are derived in a similar way asexplained above.

FIG. 1C is a block diagram illustrating a system in accordance with yetother embodiments of the present invention. Here it is illustrated howonce properties of light or radiation sources in the scene have beenderived; the newly derived data relating to the light sources can bereiterated and used in order to detect more objects in the scene basedon detected shadows. Thus for example, the properties of light orradiation sources 14A and 14B are derived based on shadows 12A-12C andobject 10 as explained above. Now, the data relating to light orradiation sources 14A and 14B can be used, together with capturedsilhouette 12D in order to detect object 10A and its properties. Theposition of object 10A may be derived based on silhouette 12D cast bylight or radiation sources 14A and 14B.

FIG. 1D is a block diagram illustrating a system in accordance with yetother embodiments of the present invention. The aforementioned scene mayfurther include one or more reflective surfaces such as mirror 40. Animage 14D of light or radiation source 14A will also illuminate thescene and serve as a radiation or light source of its own.

In a case that information regarding the reflective surfaces in thescene is provided (position and properties), it would be possible toanalyze reflections (light or radiation image source 14D) of the lightor radiation real source 14A, and/or analyze the light or radiation raysthrough the reflective surfaces 40, in order to get more informationabout light or radiation real source 14A.

For example, for real light or radiation source 14A and one object 10there is one shadow 12A. In a case that mirror 40 is known (e.g. itsequation or mere location and orientation), an auxiliary shadow 12D willalso be present. Analyzing both shadows and both real and image light orradiation sources (or back tracing their rays) will provide more insighton the properties of real light source 14A.

FIG. 2 is a drawing illustrating an aspect of a system in accordancewith embodiments of the present invention. In cross section 200A lightor radiation source 20 is shown with silhouette 24A cast on a surface26. Two objects (or more) such as 22A and 22B may be candidates for thesilhouette-light source (or radiation source) pair. This ambiguity issolved as illustrated by cross section 200B which shows an additionallight or radiation source 20B on top of 20A. Here, the composite (orcompound) silhouette 24B together with the two light or radiationsources will help identify object 22B as the only one.

FIG. 3 is a high level flowchart illustrating a method 300 for methodidentifying light or radiation sources affecting a scene, based on ananalysis of silhouettes present in the scene. Method 300 may include thestep of: sensing at least one image of a scene containing surfaces andobjects, wherein the scene is illuminated by at least one distinct lightor radiation source 310; maintaining a database of the scene whichstores approximate positions of at least some of the objects in thescene 320; identifying at least one candidate silhouette suspected to becast by the at least one distinct light or radiation source 330; andderiving properties of the at least one distinct light or radiationsource, based on the at least one identified silhouette, and furtherbased on data derived from the database 340.

According to some embodiments of the present invention, the distinctlight or radiation source is defined as a non-ambient light source.

According to some embodiments of the present invention, the deriving ofthe properties of the at least one distinct light or radiation sourcecomprises mapping the scene into a map of likelihood and confidence asto the presence and properties of the at least one distinct light orradiation source.

According to some embodiments of the present invention, the deriving ofthe light source properties is achieved by backtracking the rays oflight or radiation from outlines of the at least one candidatesilhouette 12, based on boundaries of the objects in the scene asderived from the database 120.

According to some embodiments of the present invention, the methodfurther includes a step of calculating silhouettes based on the derivedproperties of the at least one distinct light or radiation source andchecking presence of real silhouettes wherein the calculated silhouettesare located, for validating the derived properties of the at least onedistinct light source.

According to some embodiments of the present invention, the methodfurther includes a step of updating the properties of the at least onedistinct light or radiation source, based on a difference between thelocation of the calculated silhouettes and the properties of thevalidated silhouettes such as shape, color (bandwidth), location, andthe like.

According to some embodiments of the present invention, the methodfurther includes a step of detecting regions in the scene that are freefrom silhouettes, and calculating, based on data derived from thedatabase, where distinct light or radiation source cannot be located.

According to some embodiments of the present invention, the methodfurther includes a step of obtaining an ambient light or radiation levelin the scene and normalizing the derived properties of the at least onedistinct light or radiation source based on the ambient light orradiation level.

In the above description, an embodiment is an example or implementationof the inventions. The various appearances of “one embodiment,” “anembodiment” or “some embodiments” do not necessarily all refer to thesame embodiments.

Although various features of the invention may be described in thecontext of a single embodiment, the features may also be providedseparately or in any suitable combination. Conversely, although theinvention may be described herein in the context of separate embodimentsfor clarity, the invention may also be implemented in a singleembodiment.

Reference in the specification to “some embodiments”, “an embodiment”,“one embodiment” or “other embodiments” means that a particular feature,structure, or characteristic described in connection with theembodiments is included in at least some embodiments, but notnecessarily all embodiments, of the inventions.

It is to be understood that the phraseology and terminology employedherein is not to be construed as limiting and are for descriptivepurpose only.

The principles and uses of the teachings of the present invention may bebetter understood with reference to the accompanying description,figures and examples.

It is to be understood that the details set forth herein do not construea limitation to an application of the invention.

Furthermore, it is to be understood that the invention can be carriedout or practiced in various ways and that the invention can beimplemented in embodiments other than the ones outlined in thedescription above.

It is to be understood that the terms “including”, “comprising”,“consisting” and grammatical variants thereof do not preclude theaddition of one or more components, features, steps, or integers orgroups thereof and that the terms are to be construed as specifyingcomponents, features, steps or integers.

If the specification or claims refer to “an additional” element, thatdoes not preclude there being more than one of the additional element.

It is to be understood that, where the claims or specification refer to“a” or “an” element, such reference is not be construed that there isonly one of that element.

It is to be understood that, where the specification states that acomponent, feature, structure, or characteristic “may”, “might”, “can”or “could” be included, that particular component, feature, structure,or characteristic is not required to be included.

Where applicable, although state diagrams, flow diagrams or both may beused to describe embodiments, the invention is not limited to thosediagrams or to the corresponding descriptions. For example, flow neednot move through each illustrated box or state, or in exactly the sameorder as illustrated and described.

Methods of the present invention may be implemented by performing orcompleting manually, automatically, or a combination thereof, selectedsteps or tasks.

The descriptions, examples, methods and materials presented in theclaims and the specification are not to be construed as limiting butrather as illustrative only.

Meanings of technical and scientific terms used herein are to becommonly understood as by one of ordinary skill in the art to which theinvention belongs, unless otherwise defined.

The present invention may be implemented in the testing or practice withmethods and materials equivalent or similar to those described herein.

While the invention has been described with respect to a limited numberof embodiments, these should not be construed as limitations on thescope of the invention, but rather as exemplifications of some of thepreferred embodiments. Other possible variations, modifications, andapplications are also within the scope of the invention. Accordingly,the scope of the invention should not be limited by what has thus farbeen described, but by the appended claims and their legal equivalents.

The invention claimed is:
 1. A method comprising: maintaining a databaseof a scene containing objects, wherein the scene is illuminated by atleast one distinct radiation source, wherein the database storesapproximate positions of at least portions of at least some of theobjects in the scene; identifying at least one candidate silhouette, inat least one image of the scene, that is suspected to be cast by the atleast one distinct radiation source; and deriving properties of the atleast one distinct radiation source, based on the at least oneidentified silhouette, based on data derived from the database.
 2. Themethod according to claim 1, wherein a distinct radiation source is anon-ambient radiation source.
 3. The method according to claim 1,wherein the deriving of the properties of the at least one distinctradiation source comprises mapping the scene into a map of likelihoodand confidence as to the presence and properties of the at least onedistinct radiation source.
 4. The method according to claim 1, whereinthe deriving of the radiation source properties is achieved bybacktracking the radiation lines from outlines of the at least onecandidate silhouette, based on boundaries of the objects in the scene asderived from the database.
 5. The method according to claim 1, furthercomprising calculating silhouettes based on the derived properties ofthe at least one distinct light source and checking presence of realsilhouettes wherein the calculated silhouettes are located, forvalidating the derived properties of the at least one distinct radiationsource.
 6. The method according to claim 5, further comprising updatingthe properties of the at least one distinct radiation source, based on adifference between the properties of the calculated silhouettes and theproperties of the validated silhouettes.
 7. The method according toclaim 1, further comprising detecting regions in the scene that are freefrom silhouettes, and calculating, based on data derived from thedatabase, where distinct light or radiation source cannot be located. 8.The method according to claim 1, further comprising obtaining an ambientradiation level in the scene and normalizing the derived properties ofthe at least one distinct radiation source based on the ambientradiation level.
 9. The method according to claim 1, further comprisingobtaining information regarding at least one reflective surface locatedin the scene, and wherein said deriving properties of the at least onedistinct radiation source, is further based on at least one identifiedsilhouette known to be cast by reflections of the at least one distinctradiation source reflected by said at least one reflective surface. 10.A system comprising: a database of a scene containing objects, whereinthe scene is illuminated by at least one distinct radiation source,wherein the database is configured to store approximate positions of atleast portions of at least some of the objects in the scene; and acomputer processor configured to: identify at least one candidatesilhouette, in at least one image of the scene, that is suspected to becast by at least one object, and derive at least some properties of theat least one radiation source, based on the at least one identifiedsilhouette and the at least one corresponding object.
 11. The systemaccording to claim 10, wherein a distinct radiation source is anon-ambient radiation source.
 12. The system according to claim 10,wherein the deriving of the properties of the at least one distinctradiation source comprises mapping the scene into a map of likelihoodand confidence as to the presence and properties of the at least onedistinct radiation source.
 13. The system according to claim 10, whereinthe deriving of the radiation source properties is achieved bybacktracking the rays of radiation from outlines of the at least onecandidate silhouette, based on boundaries of the objects in the scene asderived from the database.
 14. The system according to claim 10, furthercomprising calculating silhouettes based on the derived properties ofthe at least one distinct radiation source and checking presence of realsilhouettes wherein the calculated silhouettes are located, forvalidating the derived properties of the at least one distinct radiationsource.
 15. The system according to claim 14, further comprisingupdating the properties of the at least one distinct radiation source,based on a difference between the properties of the calculatedsilhouettes and the properties of the validated silhouettes.
 16. Thesystem according to claim 10, further comprising detecting regions inthe scene that are free from silhouettes, and calculating, based on dataderived from the database, where distinct radiation source cannot belocated.
 17. The system according to claim 10, further comprisingobtaining an ambient radiation level in the scene and normalizing thederived properties of the at least one distinct radiation source basedon the ambient radiation level.
 18. The system according to claim 10,wherein the computer processor is further configured to obtaininformation regarding at least one reflective surface located in thescene, wherein said deriving properties of the at least one distinctradiation source, is further based on at least one identified silhouetteknown to be cast by reflections of the at least one distinct radiationsource reflected by said at least one reflective surface.